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Introduction 

The practice of forensic science has existed for centuries. Each year, 
hundreds of thousands of cases are closed, suspects cleared, and offenders 
convicted through routine, accurate, and reliable forensic testing. Forensic 
testing includes chemical analysis to determine the nature of seized drugs; 
examinations performed on physical materials such as fibers, glass, and 
spent bullet casings; and examination of biological materials such as DNA . 1 
Tests performed for each of these examinations, regardless of the materials 
examined, are strictly prescribed by laboratory policies, supported by peer- 
reviewed research, and lead to accurate and reliable results . 2 

A casual reader of recent media reports might be led to believe that 
forensic science lacks any scientific credibility . 3 However, this narrative is 
completely inaccurate and at odds with the scientific excellence that exists 
throughout the forensic science community. Forensic disciplines are 
grounded in diverse sciences such as chemistry, biology, and physics, and 
every forensic discipline practiced in an accredited forensic laboratory must 
demonstrate that it is reliable, accurate, and fit for its intended use. 
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1. See Laboratory Services, FBI, https://www.fbi.gov/services/laboratory 
[https://perma.cc/6NQP-7YQ7] (last visited Feb. 26, 2018). 

2. Id. 

3. See Eric S. Lander, Fix the Flaws in Forensic Science, N.Y. Times (Apr. 21, 2015), 
https://www.nytimes.com/2015/04/21/opinion/fix-the-flaws-in-forensic-science.html 

[https ://perma.cc/5F V8-BE4M]. 
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I. Accreditation and Its Requirements 

Accreditation and quality assurance systems assure the public that 
accredited organizations are competent and their results can be relied upon. 4 
Many groups—such as the National Commission on Forensic Science, 5 the 
National Academy of Sciences, 6 the President’s Council of Advisors on 
Science and Technology (PCAST), 7 and the Department of Justice (DOJ)— 
recognize that accreditation is critically important. In fact, in December 
2015, the Attorney General directed that all DOJ forensic laboratories must 
obtain or maintain accreditation. 8 

Accreditation is an external assessment of a laboratory’s technical 
competence to perform specific types of testing. 9 Accreditation 
demonstrates that a laboratory is performing its work correctly and 
consistent with appropriate standards. 10 To maintain this recognition, a 
laboratory is periodically reevaluated to ensure its ongoing compliance with 
accreditation requirements. 11 Laboratory accreditation is internationally 
regarded as a reliable indicator of technical competence, and it provides 
credibility and public confidence in laboratory operations. 12 An accredited 
laboratory’s quality assurance system must include written standard 
operating procedures, proficiency testing, training programs, processes for 
technical review of reports, testimony monitoring, and many other 
requirements. 13 


4. See Forensic Accreditation, ANSI-ASQ Nat’l Accreditation Bd., 
https://www.anab.org/forensic-accreditation [https://penna.cc/E4WG-DHYC] (last visited 
Feb. 26, 2018). 

5. See Nat’l Comm’n on Forensic Sci., Recommendation on Universal 
Accreditation (2015), https://www.justice.gov/archives/ncfs/file/477851/download 
[https ://penna.cc/24BX-93RG]. 

6. See Nat’l Research Council, Nat’l Acads., Strengthening Forensic Science 
in the United States: A Path Forward 6 (2009), https://www.ncjrs.gov/pdffilesl/nij/ 
grants/228091 .pdf [https://penna.ee/9Q84-RC6S], 

7. See President’s Council of Advisors on Sci. & Tech., An Addendum to the PCAST 
Report on Forensic Science in Criminal Courts, WHITE HOUSE 4 (Jan. 6, 2017), 
https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_foren 
sics_addendum_fmalv2.pdf [https://penna.cc/8A4D-57HX] (“Forensic scientists cite the role 
of professional organizations, certification, accreditation, best practices manuals, and 
training within their disciplines. PCAST recognizes that such practices play a critical role in 
any professional discipline.”). 

8. Press Release, U.S Dep’t of Justice, Justice Department Announces New 
Accreditation Policies to Advance Forensic Science (Dec. 7, 2015), https://www.justice.gov/ 
opa/pr/justice-department-announces-new-accreditation-policies-advance-forensic-science 
[https://penna.cc/YFK7-NEYL]. 

9. Forensic Accreditation, supra note 4. 

10. See id. 

11. See Laboratory Services, supra note 1. 

12. The Advantages of Being Accredited, Int’l Lab. Accreditation Cooperation 
(2015), http://ilac.org/?ddownload=898 [https://perma.cc/X9X8-ZLPZ]. 

13. See Laboratory Services, supra note 1. 
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According to the Bureau of Justice Statistics, 88 percent of the 409 
publicly-funded forensic crime labs in the United States are accredited. 14 
Unaccredited labs are often very small—less than ten people—and offer 
services in a limited number of disciplines. In addition to forensic 
laboratories, laboratories performing other types of tests are accredited 
according to the same international standard. 15 This includes 
environmental labs checking for levels of lead in groundwater, chemistry 
labs preparing chemicals for consumer use, or food labs ensuring the safety 
of our food supply. 16 

The validation of test methods is also an accreditation requirement. 
Validation is the “confirmation, through the provision of objective evidence 
(3.8.3), that the requirements (3.6.4) for a specific intended use or 
application have been fulfilled.” 17 Validation experiments are designed to 
determine whether a method yields correct results when the right answer— 
the ground truth—is known. These are empirical tests that are conducted 
prior to laboratory implementation of a method. 

Validation experiments are fundamentally different than equipment 
checks, which simply ensure that a particular piece of equipment is 
operating within defined parameters. 18 Validation may test the limitations 
of a method by analyzing a wide range of factors that are relevant and 
appropriate to a given application. 19 When validation provides insight 
regarding the limitations of a method, these limitations should be shared in 
legal proceedings. 20 The focus of validation is to confirm, through 
objective data, that the requirements for a specific intended use are 
fulfilled. 21 In contrast, method verification is the confirmation that the 
laboratory can properly use the method prior to its implementation. 22 
However, neither method validation nor verification can produce an error 
rate for an entire discipline or an individual examiner. 


14. Andrew M. Burch et al., U.S. Dep’t of Justice, Publicly Funded Forensic 

Crime Laboratories: Quality Assurance Practices, 2014, at 1 (2016), 

https://www.bjs.gov/ content/pub/pdf/pffclqapl4.pdf [https://pema.cc/MYS4-L6JS]. 

15. See ISO/IEC 17025:2017, ISO, https://www.iso.Org/obp/ui/#iso:std:iso-iec: 
17025:ed-3:vl:en [https://pema.ee/27J4-Y43U] (last visited Feb. 26, 2018). The 
International Organization for Standardization (ISO) is an independent, non-governmental 
international organization with a membership of 161 national standards bodies. All About 
ISO, ISO, https://wivw.iso.org/about-us.html [https://pema.ee/CP5P-BE7M] (last visited 
Feb. 26, 2018). The members develop voluntary, consensus-based market-relevant 
standards. Id. The ISO/IEC 17025:2017 standard focuses specifically on requirements for 
the competence, impartiality, and consistent operation of testing and calibration laboratories. 
See ISO/IEC 17025:201 7, supra. 

16. Forensic Accreditation, supra note 4. 

17. ISO 9000:2015, ISO, https://ivww.iso.Org/obp/ui/#iso:std:iso:9000:ed-4:vl:en 
[https://pema.cc/ZLW3-RG7L] (last visited Feb. 26, 2018). 

18. See Validation Information for Public DNA Laboratories, Nat’L Inst. Just., 
https://ivww.nij.gov/topics/forensics/lab-operations/Pages/validation.aspx 
[https://pema.cc/74J5-Q97C] (last visited Feb. 26, 2018). 

19. See id. 

20. See id. 

21. See id. 

22. ISO/IEC 17025:2017, supra note 15. 
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Forensic examiners must complete extensive training to be qualified to 
perform casework in accredited laboratories. 23 Training programs can be 
one to two years, or longer, and require examiners to perform analyses on 
samples with a known correct answer. 24 The examiner must also 
demonstrate a thorough understanding of the science behind the method 
employed and an understanding of lab policies, procedures, legal rules, 
evidence handling, etc. 25 The examiner must undergo oral examinations, 
mock trials, and competency tests for which the correct answer is known. 26 
In addition, all examiners must demonstrate competency to apply the 
processes accurately and reliably before they are assigned actual cases. 27 
Once qualified to conduct casework in an accredited laboratory, every 
examiner undergoes continual competency monitoring through proficiency 
tests administered at least once per year. 28 

Testimony monitoring is also a requirement for accredited laboratories. 29 
The Federal Bureau of Investigation (FBI) Laboratory requires that 
examiners request a transcript for each testimony provided. 30 FBI 
examiners also must follow approved standards for scientific testimony and 
reports, 31 which document the acceptable range of conclusions expressed in 
both laboratory reports and testimony. 32 The DOJ is developing similar 
documents called Uniform Language for Testimony and Reports, 33 as well 
as a testimony-monitoring framework, which will apply to all DOJ 
laboratories. 34 The purpose of these testimony-monitoring activities is to 
prevent examiner testimony from exceeding scientific limitations. 35 

Forensic science research plays a critical role in the culture of continuous 
improvement that is part of a rigorous quality assurance program. Such 
research seeks to develop new capabilities while providing enhancement 
and support for existing capabilities. For example, forensic science 


23. See generally Am. Soc’y Crime Lab. Dirs./Lab. Accreditation Bd., 

Supplemental Requirements for the Accreditation of Forensic Science Testing 
Laboratories (2011), 

http://des.wa.gov/sites/default/files/public/documents/About/1063/RFP/Add7Item4 
ASCLD.pdf [https://penna.cc/45LX-4NZU]. 

24. Mat 21. 

25. See id. at 11-13. 

26. Id. 

27. Id. 

28. Id. at 18. 

29. U.S. Gov’t Accountability Office, Chemistry and Trace Evidence Units 
Generally Adhere to Quality Standards, but Could Review More Examiner 
Testimonies (2017), https://www.gao.gov/assets/690/685507.pdf [https://penna.cc/RS9V- 
XHKV], 

30. Id. 

31. See Forensic Science, U.S. Dep’t Just., https://www.justice.gov/archives/dag/ 
forensic-science#prop [https://penna.cc/APD8-AZLW] (last visited Feb. 26, 2018) (denoting 
laboratory standards and reporting requirements). 

32. Id. 

33. See id. 

34. Press Release, U.S Dep’t of Justice, Justice Department Announces Plans to 
Advance Forensic Science (Aug. 7, 2017), https://www.justice.gov/opa/pr/justice- 
department-announces-plans-advance-forensic-science [https://penna.cc/56VJ-F39E]. 

35. See id. 
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research and development in the 1980s provided the groundwork for 
monumental progress in the development and advancement of DNA 
testing. 36 In addition, the FBI’s studies on latent print examinations have 
provided tremendous insight into the reliability of latent fingerprint 
examination and the decision-making process of latent print examiners. 37 
The FBI recognizes the importance of these studies and has begun similar 
studies in three pattern-comparison disciplines. 

Accreditation, validation, research, training, and testimony monitoring 
are important activities for demonstrating the reliable practice of forensic 
science. Together, they help ensure the overarching goals of finding the 
right answer, correctly communicating that answer, and continuously 
improving our ability to deliver quality results. 

II. Validation Studies 

Validation is the process used to determine whether or not a method or 
technique is fit for a given application. The PCAST Report (“Report”) 
asserts that a forensic discipline must demonstrate “foundational validity” 
and “validity as applied” for a discipline to be scientifically valid and 
reliable. 38 Flowever, the Report conflates two disparate topics in its 
discussion of “validity.” 

The authors claim that foundational validity requires the performance of 
multiple “black box” studies. 39 Flowever, black-box studies are decision- 
analysis experiments performed across a broad range of practitioners, and 
do not validate a specific methodology. 40 The authors do not offer any 
scientific basis to support their assertions. At the same time, the authors 
encourage federal judges to “take into account the appropriate scientific 
criteria for assessing scientific validity” in their “gatekeeper” role. 41 The 
authors use their unique criteria in their effort to discredit numerous 
validation studies. 42 They argue that, because the research reviewed did not 


36. See Peter Gill et al., Forensic Application of DNA Fingerprints,’ 318 Nature 577 
(1985). 

37. See generally R. Austin Hicklin et al.. Assessing the Clarity of Friction Ridge 
Impressions, 226 Forensic Set. Int’l 106 (2013); R. Austin Hicklin et al., Latent 
Fingerprint Quality: A Survey of Examiners, 61 J. FORENSIC IDENTIFICATION 385, 385-419 
(2011); Bradford T. Ulery et al., Changes in Latent Fingerprint Examiners ’ Markup Between 
Analysis and Comparison, 247 Forensic Sci. Int’l 54 (2015); Bradford T. Ulery et al., 
Measuring What Latent Fingerprint Examiners Consider Sufficient Information for 
Individualization Determinations, 9 PLOS ONE 1 (2014); Bradford T. Ulery et al., 
Repeatability and Reproducibility of Decisions by Latent Fingerprint Examiners, 1 PLOS 
ONE 1 (2012); Bradford T. Ulery et al., Understanding the Sufficiency of Information for 
Latent Fingerprint Value Determinations, 230 FORENSIC SCI. Int’l 99 (2013). 

38. President’s Council of Advisors on Sci. & Tech., Exec. Office of the 

President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of 
Feature-Comparison Methods 44-56 (2016) [hereinafter PCAST Report], 

https://obamawhiteliouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_foren 
sic_science_report_final.pdf [https://penna.cc/VJB4-5JVQ]. 

39. Id. 

40. See id. at 65-66. 

41. Id. at 41, 142. 

42. See generally id. 




44 


SCIENTIFIC EXCELLENCE 


[Vol. 86 


fit the authors’ validation paradigm, these scientific disciplines lack 
empirical evidence to support PCAST’s approval as valid science. 43 This 
position ignores much peer-reviewed research, overlooks critical aspects of 
many studies, and fails to acknowledge the empirical value of these studies. 

For example, in the firearms discipline, PCAST ignores a large number 
of studies based on their criticism of the test designs. Two studies were 
discarded because for using a “within-set comparison” design in which the 
samples were examined in a pair-wise approach. 44 The Report asserted that 
it was “impossible to estimate the false positive rate among conclusive 
examinations, which is the key measure for consideration” as the reason for 
rejecting these studies. 45 Fiowever, these same studies showed that of 1037 
different-source comparisons performed, no false identifications or false 
eliminations were reported. 46 PCAST dismissed four additional studies 
based on the use of a “closed-set” experimental design because the source 
gun was always present. 47 PCAST opined that “the closed-set design is 
problematic in principle,” and was therefore “not appropriate for assessing 
scientific validity.” 48 In these studies, the researchers utilized particularly 
challenging samples that employed consecutively-manufactured firearms. 49 
This represents the worst-case scenario for toolmarks that carry over from 
one machined part to the next. Despite these challenging samples, all of 
these studies showed that firearms examiners reliably and accurately 
associated the questioned toolmarks with the correct source. 50 In a final 
example, PCAST ignored another study due to the partly open-set design, in 
which some of the questioned samples did not have a matching known 
standard. 51 Each of these studies provide important insights into the 


43. See id. 

44. Id. at 107 & nn. 3 19-20; see also Charles S. DcFrance & Michael D. Van Arsdale, 
Validation Study of Electrochemical Rifling, 35 Ass’n Firearm & Toolmark Examiners J. 

1 (2003); Erich D. Smith, Cartridge Case and Bullet Comparison Validation Study with 
Firearms Submitted in Casework, 36 Ass’n Firearm & Toolmark Examiners J. 130 
(2005). 

45. PCAST Report, supra note 38, at 107. 

46. Id. at 107 n.320. 

47. Id. at 108-09 & nn.324-28; see also David J. Bmndage, The Identification of 
Consecutively Rifled Gun Barrels, 30 Ass’n Firearm & Toolmark Examiners J. 438, 438- 
44 (1998); Thomas G. Fadul Jr. et al., An Empirical Study to Improve the Scientific 
Foundation of Forensic Firearm and Tool Mark Identification Utilizing Ten Consecutively 
Manufactured Slides, 45 Ass’n Firearm & Toolmark Examiners J. 376, 376-93 (2013); 
James E. Hamby et al., The Identification of Bullets Fired from Ten Consecutively Rifled 
9mm Ruger Pistol Barrels: A Research Project Involving 507 Participants from Twenty 
Countries, 41 Ass’n Firearm & Toolmark Examiners J. 99, 99-100 (2009); Angela 
Stroman, Empirically Determined Frequency of Error in Cartridge Case Examinations 
Using a Declared Double-Blind Format, 46 ASS’N FIREARM & TOOLMARK Examiners J. 
157, 157-75 (2014). 

48. PCAST Report, supra note 38, at 109. 

49. See supra note 47 and accompanying text. 

50. See supra notes 44—49 and accompanying text. 

51. PCAST Report, supra note 38, at 108-09 & n.326; see also Thomas G. Fadul Jr. 
ET AL., MlAMl-DADE POLICE DEP’T CRIME LAB., An EMPIRICAL STUDY TO IMPROVE THE 
Scientific Foundation of Forensic Firearm and Tool Mark Identification Utilizing 
Consecutively Manufactured Clock EBIS Barrels with the Same EBIS Pattern 
(2013), www.ncjrs.gov/pdffilesl/nij/grants/244232.pdf[https://penna.cc/6UKW-6FYF]. 
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science of firearms analysis and additional empirical support for the validity 
of the discipline. 


III. Error Rate Studies 
Should Not Be Mistaken for Validation 

The Report claimed that “the foundational validity of subjective methods 
can be established only through empirical studies of examiner’s 
performance . . . such studies are referred to as ‘black-box’ studies.” 52 
Black-box studies are a way to analyze the decisions made by a range of 
examiners under defined conditions. 53 However, black-box studies should 
not be mistaken as a way to establish the validity of a specific method or the 
error rate for an entire discipline. The entire body of research and testing 
relative to a particular forensic method provides support for its scientific 
validity—not simply the number of black-box studies performed. 

Some forensic commentators conflate distinct issues when criticizing the 
reliability of forensic disciplines. 54 These issues concern whether: (1) a 
forensic discipline is scientifically valid and based upon sound scientific 
principles; (2) individual practitioners can identify the right answer when 
that answer is known (personal error rate); (3) different practitioners obtain 
the same answer when reviewing the same materials and data; and (4) there 
is a universal error rate for a specific discipline. 55 The Report focuses on 
the fourth issue, 56 which has nothing to do with the method validity, but 
rather the decisions made by examiners under a defined range of conditions. 
Furthermore, it is problematic when scientific validity is confused with the 
legal standard for admissibility. Unfortunately, the Report only exacerbates 
the confusion. 

Regarding microscopic hair analysis, PCAST discussed a 2002 paper by 
Max M. Houck and Bruce Budowle. 57 The study found that in 11 percent 
of cases in which hairs were microscopically associated, DNA analysis 
revealed that the samples originated from different individuals. 58 
Unfortunately, many misinterpret the results of this study to mean that 
microscopic hair comparison has an 11 percent error rate. PCAST correctly 
noted that these associations may not have been incorrect but, instead, were 
simply characteristics that were shared by chance. 59 Because microscopic 
hair comparison involves class-level associations, hair cannot be used as a 


52. PCAST Report, supra note 38, at 49. 

53. Id. 

54. See generally Kelly Pyrek, Forensic Science Under Siege: The Challenges of 
Forensic Laboratories and the Medico-Legal Investigation System (2007). 

55. See id. at 241-43 (discussing the need to disclose errors, error rates, and sources of 
errors in forensic science experiments to maintain confidence in the scientific integrity of the 
results). 

56. See generally PCAST Report, supra note 38. 

57. Id. at 28 & n.33; see also Max M. Houck & Bruce Budowle, Correlation of 
Microscopic and Mitochondrial DNA Hair Comparisons, 47 J. FORENSIC SCIS. 1, 1-4 
( 2002 ). 

58. PCAST Report, supra note 38, at 28. 

59. Id. 
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unique identifier. 60 One key point of the Houck and Budowle study that 
PCAST did not discuss was the combined power of discrimination by the 
joint use of microscopic and mitochondrial DNA analysis. 61 Instead, the 
Report characterized the false positive rate for microscopic hair comparison 
in the study as applicable to the technique in general. 62 

Conclusion 

Science continuously evolves and is built upon observation, study, and 
experience that spans hundreds of years. The justice system would not be 
well served by the exclusion of reliable forensic methods and techniques 
that provide valuable information to a wide range of stakeholders. Critical 
reviews of past and current practices assist in the continual process of 
evaluation and improvement. However, they do not invalidate the entire 
body of past scientific research and achievement. 63 


60. Class-level evidence encompasses a group of objects or persons with characteristics 
that are shared by the group. The characteristics are not unique to a particular object or 
person but serve to place the evidence into a smaller group of objects. 

61. Houck & Budowle, supra note 57, at 2-4. See generally PCAST Report, supra 
note 38. 

62. See generally PCAST Report, supra note 38. 

63. See Ernst Mayr, The Growth of Biological Thought: Diversity, Evolution, 
and INHERITANCE 831 (1982) (“All interpretations made by a scientist are hypotheses, and 
all hypotheses are tentative. They must forever be tested and they must be revised if found 
to be unsatisfactory. Hence, a change of mind in a scientist, and particularly in a great 
scientist, is not only not a sign of weakness but rather evidence for continuing attention to 
the respective problem and an ability to test the hypothesis again and again.”). 




